SDD

University of London CM 2010 Software Design and Development

---

Project maintained by ccy05327 Hosted on GitHub Pages — Theme by mattgraham

Topic 4 Testing

• Requirements
• Test plans
• Usability testing
• Accessibiltiy

Week 13 User stories: what should they be able to do?

Learning objectives

• Explain why requirements are necessary and who the stakeholders are
• Name several methods that people use to write requirements
• Use the ‘Easy Approach to Requirements Syntax’ (EARS) to write requirements

Reading

• ✓ PDF Gregory, S.’The Unplanned Journey of a Requirements Engineer in Industry: An Introduction’, IEEE Software 34(5) 2017, pp.16-19.

• ✓ PDF Mavin, A., P. Wilkinson, A. Harwood and M. Novak ‘Easy Approach to Requirements Syntax (EARS)’, 2009 17th IEEE International Requirements Engineering Conference (Atlanta, GA: IEEE, 2009), pp.317-322.

• ✓ PDF R. L. Glass ‘An ancient (but still valid?) look at the classification of testing’, IEEE Software 2(6) Nov-Dec 2008, pp.112-112.

• ✓ PDF R. L. Glass ‘A classification system for testing, part 2’, IEEE Software 26(1) Jan-Feb 2009, pp.104-104.

• ✓ PDF Chernak, Y. ‘Validating and improving test-case effectiveness’, IEEE Software 18(1) Jan-Feb 2001, pp.81-86.

Intro to Requirements

How to ensure that a system does what it is supposed to do?

1. Describe what is is supposed to do (requirements)
2. Check if it meets the requirements (testing)

Who are requirements for?

1. End-users
2. Producing party
3. Perchasing party
4. Developers

What kind of requirements do they understand?

Requirements techniques

Natural Language

[Constrained natural language] Simplified technical English:

• Short sentences
• Short paragraphs
• Limit grammar
• Active voice

[Graphical] UML (Unified Modeling Langauge): Generate a UML diagram from classes

[Formal] Z Language: a formal specification language used for describing and modelling computing systems.

EARS

Easy Approach to Requirements Syntax

---

Week 14 Black box and white box testing

Learning objectives

• Write out step by step and matrix style test procedures
• Differentiate between black box and glass/white box testing and provide examples of each
• Explain how automated blackbox testing can be achieved in video games

Definition

Black box

1. system or component whose inputs, outputs, and general function are known but whose contents or implementation are unknown or irrelevant.

2. pertaining to an approach that treats a system or component as in (1).

Glass or white box – unit testing

1. system or component whose internal contents or implementation are known.

2. pertaining to an approach that treats a system or component as in (1).

Requirements can be written for both or one.

Testing

Activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component.

test case specification: document specifying inputs, predicted results, and a set of execution conditions for a test item.

test procedure specification: document specifying a sequence of actions for the execution of a test.

There is an ISO standard for software testing documentation.

Test procedure template

• ISO Test procedure template matrix

• ISO Test procedure template step by step

Automated black box testing

Replying exclusively on playtesting conducted by humans can be costly and inefficient. Artificial agents could perform much faster play sessions, allowing the exploration of much more of the game space in much shorter time.

Zhao et al. “Winning isn’t everything: Training agents to playtest modern games.”

This is the paper describing the Sims Mobile agent:

• PDF Zhao, Y., I. Borovikov, A. Beirami, J. Harder, J. Kolen, J. Pestrak, J. Pinto, R. Pourabolghasem, H. Chaput, M. Sardari et al. ‘Winning isn’t everything: Training agents to playtest modern games’, AAAI Workshop on Reinforcement Learning in Games 2019.

This is a very detailed paper describing the state of the art in automated video game testing of video games. Read this if you want to see how one might go about analysing a research field, and also to find out what is possible in the area of automated testing.

• PDF Albaghajati, A.M. and M.A.K. Ahmed ‘Video game automated testing approaches: an assessment framework’, IEEE Transactions on Games.

---

Week 15 Usability testing

Learning objectives

• Explain hwo the terms efficient, effective and satisfying relate to usability
• Give an example of a standard usability survey and use it to evaluate usabilitly
• Evaluate the usability of a piece of software against Nielsen’s usability principles

Reading

• PDF Ferre, X., N. Juristo, H. Windl and L. Constantine ‘Usability basics for software developers’, IEEE Software 18(1) 2001, pp.22-29.

• PDF Brooke, J. ‘SUS: a retrospective’, Journal of Usability Studies, 8(2) 2013.

• PDF Lewis, J.R., B.S. Utesh, D.E. Maher ‘UMUX-LITE: when there’s no time for the SUS’, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2013, pp.2099-2102.

• PDF Cherry, E. and C. Latulipe ‘Quantifying the creativity support of digital tools through the creativity support index’, ACM Transactions on Computer-Human Interaction 21 2014.

• PDF Molich, R. and J. Nielsen ‘Improving a human-computer dialogue’, Communications of the ACM 33(3) March 1990, pp.338–348.

Usability

the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

ISO 9241-11

Effectiveness

Whether they are able to complete a task.

Efficiency

How quickly and how easily and how fluid to complete a task.

Satisfaction

Feeling when they are using it.

Usability metrics

System Usability Scale (SUS) [1996]: A list of 10 questions to ask a user after they’ve tried the system.

Each question has a rating of strongly disagree to strongly agree (score 1 to 5 or 5 to 1)
1. I think that I would like to use the system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to use this system
5. I found the various functions of this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system.

The usability metric for user experience (UMUX) [2013]: Using less questions than SUS, similar outcome.

Each question has a rating of strongly disagree to strongly agree (score 1 to 7 or 7 to 1)
1. [This system's] capabilities meet my requirements.
2. Using [this system] is a frustrating experience.
3. [This system] is easy to use.
4. I have to spend too much time correcting things with [this system].

Creativity Support Index [2014]: the evaluation of software tools for doing creative work.

Collaboration
Enjoyment
Exploration
Expressiveness
Immersion
[Gained] Results Worth [the] Effort

Usability principles

Nielson’s 10 principles (1989)

1. Visibility of system status
   Do we know the status of the system?
   Is the piece of software communicating to us?
   What it’s ready to do and where it’s at?
2. Match between system and the real world
   A bit old. Digital correlation with the actual physical world.
3. User control and freedom
   What can I do at any point in the software?
   Is the software constraining me?
4. Consistency and standards
   UI/UX related
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
   Mentioned in the previous section
8. Aesthetic and minimalist design
   A bit subjective, also UI/UX related
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation

---

Week 16 Accessibility testing

Learning objectives

• Differentiate between accessibility and usability in terms of the target users
• User accessibility testing tools to identify accessibility problems with software systems
• Be aware that in the UK at least, there are legal requirements to make certain requirements to make certain software systems as accessible as reasonably possible

Reading

• A​Checker Web Accessibility Checker

• The Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018

Accessibility

extent to which products, systems, services, environments and facilities can be used by people from a population with the widest range of characteristics and capabilities to achieve a specified goal in a specified context of use

ISO/IEEE 2017 p6, 3.41

Usability

A subset of accessibility. Giving a certain group of users access

Accessibility

Giving as many people as possible access

Legal requirements in the UK

1. Subject to regulation 7, public sector bodies must comply with the accessibility requirement.

The Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018

accessibility requirement means the requirement to make a website or mobile application accessible by making it perceivable, operable, understandable, and robust.

---